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EXECUTIVE  SUMMARY 


This  report  summarizes  the  Jefferson  Proving  Ground  (JPG)  Technology  Demonstrations  (TD) 
Program  conducted  between  1994  and  1999.  These  demonstrations  examined  the  current  capability 
of  commercial  and  military  equipment  to  detect,  classify,  and  remove  Unexploded  Ordnance  (UXO). 
No  baseline  performance  standards  existed  for  UXO  systems  prior  to  the  JPG-TD. 

Phase  I  was  conducted  in  1994.  The  objectives  were  to  evaluate  existing  and  promising 
technologies  for  detecting  and  remediating  UXO.  Phase  I  results  showed  less  than  expected  detection 
and  localization  capability  and  no  capacity  to  discriminate  between  ordnance  and  non-ordnance. 
The  average  probability  of  detection  (Pd)  was  0.62.  Airborne  systems  were  unable  to  detect  more 
than  8%  of  the  UXO.  Remediation  vendors  excavated  4  to  11  targets  per  40  hours  on  site.  At  the 
completion  of  the  Phase  I  testing  at  JPG,  Congress  requested  a  demonstration  of  the  most  applicable 
technologies  at  five  “Live  Site”  locations  to  allow  collection  of  comparison  data  in  a  variety  of 
vegetative  and  geologic  conditions.  The  average  Pd  for  the  Live  Sites  demonstration  was  0.44. 

The  Phase  II  effort  in  1995  provided  an  opportunity  to  demonstrate  capabilities  based  on  lessons 
learned  in  Phase  I.  Phase  II  showed  a  significant  improvement  in  Pd  from  0.62  to  0.68.  The  better 
performers  in  Phase  II  detected  over  80  percent  of  the  ordnance,  but  they  also  reported  four  to  twenty 
times  more  targets  as  ordnance  (false  alarms). 

Phase  III  was  conducted  in  1996.  This  phase  provided  a  more  homogeneous  set  of  buried  UXO 
by  simulating  typical  range  scenarios,  i.e.  aerial  gunnery  range,  an  artillery  and  mortar  range,  a 
grenade  and  submunition  range,  and  an  interrogation  and  burial  area.  JPG  Phase  III  results  showed 
that  state  of  the  art  technology  exists  that  is  capable  of  detecting  a  substantial  portion  of  the 
ordnance.  To  date,  no  vendor  demonstrated  the  capability  to  discriminate  ordnance  and  non¬ 
ordnance,  although  average  probabilities  of  detection  increased  during  successive  phases  from  62% 
to  77%.  However,  the  false  alarm  rate  increased. 

Phase  IV  was  conducted  in  1998.  The  purpose  was  to  demonstrate  the  capabilities  of  technology 
to  discriminate  between  UXO  and  non-UXO.  Results  of  Phase  IV  show  there  is  a  developing 
capability  to  distinguish  between  ordnance  and  non-ordnance.  Five  of  the  ten  demonstrators  showed 
a  capability  to  discriminate  ordnance  and  non-ordnance  over  50%  of  the  time.  During  the  history  of 
the  JPG  Technology  Demonstrations,  76  characterization  and  remediation  technologies  were 
demonstrated.  These  efforts  provided  Government  and  Industry  with  a  unique  method  of  learning 
and  applying  knowledge  about  UXO  Clearance  in  a  controlled  field  environment.  Phases  I  through 
III  demonstrated  the  capabilities  of  current  technology  as  it  applies  to  the  detection  of  ordnance 
while  Phase  IV  demonstrated  the  state-of-the-art  for  current  UXO  discrimination  capabilities.  The 
ability  to  detect  and  discriminate  decreases  both  the  risk  and  cost  of  UXO  site  remediation.  It  is 
strongly  recommended  that  these  technology  demonstrations  be  continued.  There  is  a  need  to 
document  the  capabilities  and  limitations  of  sensor  systems  as  they  are  developed.  Consideration 
should  be  given  to  designating  other  DOD  areas  for  technology  demonstrations  to  provide  geological 
and  vegetative  variation. 

As  a  performance  test,  the  JPG  demonstrations  have  shown  the  strengths  of  some  sensors,  the 
weaknesses  of  others,  and  provided  insights  into  the  complicated  issues  surrounding  UXO  detection 
and  cleanup.  Continued  support  of  UXO  technology  demonstrations  will  significantly  reduce  site 
remediation  costs  in  the  future. 
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INTRODUCTION 

This  report  documents  the  Jefferson  Proving  Ground  (JPG)  Technology  Demonstrations 
(TD)  Program  conducted  between  1994  and  1999.  These  demonstrations  examined  the  current 
capability  of  commercial  and  military  equipment  to  detect,  classify,  and  remove  Unexploded 
Ordnance  (UXO)  placed  in  various  underground  locations  simulating  typical  UXO  Clearance 
environments.  This  program  was  sponsored  by  the  Army  Environmental  Center  (AEC)  and 
executed  by  the  Naval  Explosive  Ordnance  Technology  Division  (NAVEODTECHDIV)  as  part 
of  the  Army  Environmental  Technology  program  under  congressional  direction. 

At  the  start  of  the  JPG  TD  Program,  UXO  site  characterization  efforts  were  typically 
conducted  by  retired  military  Explosive  Ordnance  Disposal  Technicians  using  “Mag  and  Elag” 
operations  conducted  UXO  Clearance  operations.  “Mag  and  Elag”  operations  consist  of  an 
operator  using  a  hand  held  magnetometer  or  an  electro-magnetic  inductance  device  to  detect  an 
underground  anomaly.  They  then  place  a  flag  in  the  ground  corresponding  to  the  center  of  the 
anomalous  signal.  The  training  and  proficiency  of  the  operator  directly  influenced  the  percentage 
of  buried  UXO  detected  and  the  number  of  anomalies  detected  that  were  not  UXO.  On  average, 
for  every  UXO  item  that  was  detected  and  excavated,  100  non-UXO  items  were  detected  and 
excavated.  “Mag  and  Elag”  is  still  the  most  commonly  used  detection  system.  The  U.S.  Army 
Corps  of  Engineers  calculated  that  75%  of  the  cost  of  UXO  Clearance  operations  are  incurred 
excavating  non-UXO  items. 

As  the  number  of  acres  of  Closed,  Transferred,  Transferring,  Active  and  Inactive  Ranges 
found  to  have  UXO  increased,  the  interest  in  reducing  the  cost  of  UXO  Clearance  operations  also 
increased.  Sensor  manufacturers,  university  and  government  researchers  proposed  a  number  of 
different  methods  for  detecting  and  classifying  buried  UXO.  While  many  claims  were  made 
about  the  effectiveness  of  magnetometers,  gradiometers,  time  domain  inductance,  frequency 
domain  inductance,  ground  penetrating  radar,  synthetic  aperture  radar,  hyperspectral  imaging, 
thermal  imaging,  seismic  detectors  and  chemical  detectors  based  on  their  performance  in 
detecting  other  buried  objects  (everything  from  mines  to  pipelines  to  mineral  deposits),  there  was 
little  documented  evidence  to  assist  the  UXO  Clearance  community  in  selecting  methods  other 
than  “Mag  and  Elag”. 

The  purpose  of  the  JPG  TD  Program  was  to  establish  a  comparative  performance  base  for 
current  technologies  and  systems.  The  program  was  not  intended  to  compare  the  performance  of 
one  specific  magnetometer  with  another  but  instead  to  answer  whether  a  technology  approach, 
for  example  magnetometry  was  currently  more  mature  than  ground  penetrating  radar. 

The  program  was  performed  in  four  phases  that  built  on  the  knowledge  and  understanding 
of  sensor  performance,  post  processing  capability  and  suitable  field  simulation  experience  gained 
from  the  previous  phase. 
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Phase  I:  Evaluate  existing  and  promising  UXO  technologies  with  emphasis  on  detection  and 
removal  of  UXO. 

Phase  lA:  Evaluation  of  best  performing  technologies  from  Phase  I  at  five  geologically  diverse 
live  sites  containing  UXO. 

Phase  n:  Provide  industry  the  opportunity  to  demonstrate  their  capabilities  based  on  the  lessons 
learned  from  Phase  I. 

Phase  ni:  Provide  a  more  homogeneous  set  of  buried  UXO  targets  based  on  Phase  II  results 
using  four  different  range  scenarios. 

Phase  rV:  Investigate  capabilities  to  discriminate  between  ordnance  and  non-ordnance  and  to 
provide  more  information  about  target  size,  type,  shape,  depth,  and  orientation. 


SUMMARY  OF  TECHNOLOGY  DEMONSTRATIONS 

No  baseline  performance  standards  existed  for  UXO  detection,  discrimination,  and 
removal  capabilities  of  contractor  or  government  systems  prior  to  the  Jefferson  Proving  Ground 
Technology  Demonstrations  (JPG-TD).  There  was  no  verified  data  that  allowed  an  installation 
manager  to  determine  what  type  of  technology  was  most  useful  for  their  UXO  Clearance  needs. 
A  proposal  by  the  Army  Environmental  Center  (AEC)  and  the  Naval  Explosive  Ordnance 
Technology  Division  (NAVEODTECHDIV)  was  funded  by  Congress  to  determine  the  current 
“State  of  Technology”  as  it  pertained  to  the  UXO  removal  problem.  To  accomplish  this,  a  Base 
Realignment  and  Closure  (BRAC)  facility  was  chosen  (JPG)  to  provide  a  “blind  test  site”. 

Two  parcels  of  property,  80  and  40  acres  respectively,  were  seeded  with  inert  ordnance 
and  non-ordnance  targets  at  depths  and  penetration  angles  of  typical  bombs,  projectiles,  mortars, 
and  rockets.^  The  goal  of  the  demonstration  was  to  document  the  capability  of  diverse 
technologies  to  detect,  identify  and  remediate  UXO.  These  demonstrations  were  not  designed  to 
provide  rigorous  scientific  comparison  of  each  sensor,  algorithm  or  excavator.  The  information 
gathered  provides  evidence  of  the  applicability  of  one  class  of  technology  versus  another.  The 
data  also  provided  a  guide  for  future  basic  and  applied  research  efforts. 

PHASE  I 

Phase  I  Demonstration  Design 

Phase  I  of  the  JPG-TD  was  conducted  in  1994.  The  objective  of  Phase  I  was  to  evaluate 
existing  and  promising  technologies  for  detecting  and  remediating  UXO.  Targets  were  buried 
(the  Government  protected  the  data  on  type,  number,  and  locations)  singly  and  in  groups.  A  wide 

*  Conventional  Weapons  Effects  Program  (CONWEP),  a  computer  program  that  can  predict  projectile  penetration 
depths.  It  is  available  to  U.S.  government  agencies  from  the  U.S.  Army  Waterways  Experiment  Station,  ATTN; 
CEWES-SS-R,  3909  Halls  Eerry  Road,  Vicksburg,  Mississippi  39180,  commercial  (601)  634-3668. 
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range  of  burial  depths  for  each  class  of  ordnance  was  used,  never  exceeding  the  maximum 
theoretical  penetration  depth  of  any  ordnance  item^Refer  to  Graph  1  and  Table  2.Twenty-six 
technologies  representing  magnetometry,  electromagnetic  inductance,  ground  penetrating  radar, 
synthetic  aperture  radar  and  infrared  sensors  were  flown,  driven  and  dragged  over  the  test  areas. 
Twenty  different  vehicle  mounted  and  man  portable  technology  demonstrators  were  required  to 
operate  on  the  40-acre  site  (16  hectare  site:  1  hectare  =  2.47  acres),  and  six  airborne  systems  were 
required  to  operate  on  the  80-acre  site  (32  hectare).  Demonstrators  were  to  provide  x,y  location 
in  Universal  Transverse  Mercator  coordinates  (UTM),  estimated  depth  of  the  detected  anomaly, 
type  of  anomaly,  and  the  attitude  (azimuth  and  inclination)  of  the  anomaly. 

The  26  demonstrated  technologies  were  evaluated  according  to  the  probability  of 
detection  (Pd=  ordnance  detected/ordnance  buried),  false  alarm  rate  (FAR=declarations  not 
matched  to  baseline  targets/area  surveyed),  target  location  error  (average  radial  distance  between 
demonstrator  declaration  and  baseline  target  location),  discrimination  between  ordnance  and  non¬ 
ordnance  (Rord  and  Rnonord),  and  various  production  rates  (time  to  excavate  target,  volume  of  earth 
removed  etc).  Some  targets  were  buried  next  to  each  other  in  order  to  test  the  resolution 
capabilities  of  various  sensors.  Three  remediation  demonstrators  were  required  to  navigate  and 
dig  up  buried  targets.  In  addition,  they  were  evaluated  on  production  rate  and  accuracy. 


Results  of  Phase  I 

Results  from  Phase  I  showed  less  than  expected  detection  and  localization  capability  for 
all  technologies.  The  probability  of  detection  for  the  single  demonstrator  that  detected  over  50% 
of  the  baseline  (buried)  ordnance  was  62%.  The  average  false  alarm  rate  (false  alarms  per 
hectare)  was  149  or  approximately  60  false  alarms  per  acre.  This  high  false  alarm  rate  was  not 
expected  since  the  area  had  been  searched  with  hand  held  magnetometers  and  most  of  the 
shrapnel,  range  targets,  and  other  ordnance  debris  normally  found  were  removed. 

No  capacity  to  discriminate  between  ordnance  and  non-ordnance  was  reported  by 
participants.  In  addition,  there  was  no  capability  to  resolve  targets  buried  close  to  each  other  or 
to  determine  azimuth  and  inclination/declination  headings.  Electromagnetic  sensors  such  as 
magnetometers,  gradiometers,  and  electromagnetic  induction  coils  proved  to  be  the  most 
effective  tools  in  detecting  UXO.  Ground  penetrating  radar  (GPR),  infrared  imaging  (IR), 
acoustics,  and  other  imaging  technologies  were  unsuccessful  in  this  phase.  Airborne  systems, 
regardless  of  sensors  used,  were  unable  to  detect  more  than  8%  of  the  buried  ordnance. 
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GRAPH  1  Theoretical  Ordnance  Penetration  Into  Soil  ^ 
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Poor  detection  performance  during  Phase  I  signaled  a  physics  limitation  of  current  sensor 
technology  in  relation  to  ordnance  burial  depth.  Future  phases  attempted  to  decrease  the  sensor 
to  ordnance  distance  to  facilitate  detection  performance  at  depths  that  were  more  representative 
of  current  UXO  Clearance  goals. 

Remote  remediation  or  removal  tasks  were  found  to  occur  at  a  rate  of  approximately  4  to 
1 1  targets  per  40  hours  on  site.  These  remediation  times  included  travel  from  the  staging  point  to 
the  dig  point,  travel  between  dig  points,  and  mechanical  breakdown  times.  This  compares  with 
0.1 -1.0  hours  per  target  for  hand  or  backhoe  excavation  (depending  on  depth). 

At  the  completion  of  the  Phase  I  testing  at  JPG,  Congress  requested  a  demonstration  of 
the  most  applicable  technologies  at  five  “Live  Site”  locations.  These  “Live  Sites”  allowed 
collection  of  comparison  data  in  geological  conditions  much  different  from  JPG.  Some  of  the 
sensor  technologies  were  thought  to  have  limited  capability  in  the  moist  clay  soil  encountered  in 
Indiana,  so  some  of  the  less  successful  technologies  at  JPG  were  also  included.  The  “Live  Sites” 
selected  were: 

Jefferson  Proving  Grounds,  IN 
Eglin  Air  Force  Base,  FL 
Yuma  Proving  Grounds,  AZ 
Fort  Jackson  Military  Reservation,  SC 
McChord  Air  Force  Base,  WA 
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Table  1  summarizes  results  from  the  live  site  demonstrations.  Average  probability  of  detection 
(Pd)  for  the  live  sites  is  0.44.  Randomly  selected  suspected  targets  were  chosen  for  excavation 
and  because  the  validation  period  lasted  only  3  to  4  weeks,  a  valid  statistical  sample  was  not 
possible.  Magnetometry  and  electromagnetic  inductance  were  by  far  the  top  performing 
technologies.  Hand  held  systems  continued  to  show  the  best  results.  Airborne  systems 
continued  to  perform  poorly. 

Table  1  Live  Sites  Probability  of  Detection 


Vendor 

Sensors 

JPG 

Eglin 

AFB 

YPG 

Ft. 

Jackson 

McChord 

AFB 

Ground 

Platforms: 

Pd 

Pd 

Pd 

Pd 

Pd 

ADI 

Magnetometer 

0.71 

0.69 

Coleman 

GPR  &  EM* 

0.54 

0.55 

Chemrad 

Magnetometer 

0.14 

Geo- 

Centers 

Magnetometer 

0.60 

0.45 

Metratek 

GPR  &  EM 

0.30 

0.61 

Vallon 

Magnetometer 

0.74 

Aerial 

Platforms: 

Aerodat 

Magnetometer 

0.07 

SRI 

GPR 

0.00 

0.00 

GPR-ground  penetrating  radar,  EM-electromagnetic  induction 
Pd-  probability  of  detection 


Phase  II  Demonstration  Design 

Phase  n  of  the  JPG-TD  was  conducted  in  1995.  The  objective  of  Phase  n  was  a 
continuation  of  the  work  started  in  Phase  I.  Phase  II  provided  industry  an  opportunity  to 
demonstrate  their  capabilities  based  on  lessons  learned  in  Phase  I.  Multiple  targets  that  were 
buried  in  the  same  excavation  the  year  before  were  removed  or  replaced  by  single  targets.  Phase 
I  had  shown  that  no  sensors  or  sensor  processing  could  resolve  multiple  targets  placed  close 
together.  For  Phase  II  a  new  40-acre  subset  of  the  80  acre  site  was  used  to  perform 
demonstrations.  Demonstrators  that  were  chosen  for  Phase  n,  who  had  demonstrated  during 
Phase  I,  were  placed  on  this  second  site.  The  same  evaluation  criteria  used  for  Phase  I  was  used 
for  Phase  II. 

Results  of  Phase  II 

Phase  n  showed  a  significant  improvement  in  Pd  from  0.62  to  0.68.  The  better 
performers  in  Phase  II  detected  over  80  percent  of  the  ordnance,  but  they  also  reported  four  to 
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twenty  times  more  targets  as  ordnance  (false  alarms).  Demonstrators  who  used  a  combination  of 
electromagnetic  induction  and  magnetometry  based  sensors  provided  the  best  performance.  The 
ability  of  industry  to  increase  detection  rates  from  Phase  I  to  Phase  n  was  attributed  to  proper 
selection  of  sensors,  systems,  ordnance  declarations,  and  past  experience  at  the  site.  This  was 
accomplished  by  industry  with  minimal  investment  by  the  Government  in  research  and 
development.  The  performers  at  this  phase  continued  to  demonstrate  an  inability  to  distinguish 
ordnance  from  the  non-UXO  targets.  This  deficiency  translates  into  a  major  cost  factor  in 
remediating  UXO  properties.  Metrics  such  as  “acres  covered  per  hour”  and  “target  localization 
error”  actually  increased  and  decreased  respectively.  This  meant  that  the  detection  contractors 
were  covering  more  acreage  but  it  may  be  harder  to  relocate  and  excavate  the  targets. 

Remediation  demonstrators  showed  roughly  the  same  production  rates  as  the  year  before 
but  with  better  navigational  accuracy.  Excavation  demonstrations  of  remotely  operated  systems 
were  demonstrated  during  both  phases.  Excavators  could  unearth  ordnance  at  only  a  fractional 
rate  (<5%)  of  how  fast  demonstrators  could  detect  it. 

Phase  III  Demonstration  Design 

Phase  ni  of  the  JPG-ATD  was  conducted  in  1996.  Phase  in  provided  a  more 
homogeneous  set  of  buried  UXO  by  simulating  four  different  range  scenarios.  These  scenarios 
were  based  on  experience  gained  by  the  U.  S.  Army  Corps  of  Engineers  in  actual  UXO  Clearance 
operations.  This  experience  showed  that  UXO  is  found  in  homogenous  groups  on  ranges,  rather 
than  diverse  groups  and  burial  depths.  The  past  phases  may  not  have  demonstrated  the  full 
capability  of  certain  technology  types  to  meet  specific  UXO  Clearance  needs.  To  gather  data  on 
specific  capabilities,  an  aerial  gunnery  range,  an  artillery  and  mortar  range,  a  grenade  and 
submunition  range,  and  an  interrogation  and  burial  area  were  set  up  on  ten-acre  grids.  Industry 
was  able  to  propose  work  on  one  or  more  ranges,  depending  upon  their  confidence  in  finding 
various  classes  of  ordnance.  Targets,  other  than  bombs,  deeper  than  2  meters  were  either  moved 
to  shallower  depths  or  removed  from  the  site  completely. 


^  U.S.  Army  Environmental  Center  and  the  Naval  Explosive  Ordnance  Disposal  Technology  Division,  Report  No. 
SEIM-AEC-ET-CR-9701 1,  UXO  Technology  Demonstration  Program  at  Jejferson  Proving  Ground,  Phase  III, 
April  1997,  http;//aec-www.apgea.army.mil:8080/ 
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TABLE  2  Phase  I-III  Buried  Ordnance 


Buried  Ordnance 

JPG-ATD 

Burial  Depth 

Phase 

(meters) 

20mm  &  30mm  (aircraft) 

I,  II 

.05  to  .30 

Mortars  (60mm  -  4.2”) 

I,  II 

.01  to  1.43 

Projectiles  (76mm  -  8”) 

I,  II 

.22  to  3.66 

General  Purpose  Bombs  (250  -  2,000  lb.) 

i,n 

.15  to  6.1 

Rockets  (2.75”  -  5”) 

i,n 

.15  to  2.44 

Aerial  Gunnery  Range  (2.75”  -  2,000  lb.) 

m 

<  3 

Artillery  and  Mortar  Range  (60mm  -  8”) 

m 

<  1.2 

Grenade  and  Submunition  Range 

m 

<0.5 

Interrogation  and  Burial  Site  (all  the  above) 

m 

<2.0 

Results  of  Phase  III 

JPG  Phase  in  focused  on  developing  relevant  performance  data  of  technologies  used  in 
site  specific  situations  to  search,  detect,  characterize,  and  excavate  UXO.  The  JPG  Phase  IH 
results  showed  that  state  of  the  art  technology  exists  (still  based  on  magnetometers  and 
electromagnetic  induction)  that  is  capable  of  detecting  a  substantial  portion  of  the  ordnance 
emplaced  for  the  four  scenarios.  Results  also  showed  that  average  probability  of  detection 
decreases  as  a  function  of  ordnance  type  and  size.  Table  3  shows  this  for  the  three  range  survey 
scenarios. 


TABLE  3  Probabilities  of  Detection  for  Typical  Range  Scenarios 


Survey  Scenario 

Range  of  Probabilities  of 

Detection  (Pd) 

Average 

Aerial  Gunnery  Range 

.23  to  1.00 

.75 

Artillery/Mortar  Range 

.03  to  .97 

.69 

Grenade  &  Submunition  Range 

.01  to  .95 

.64 

Typically,  on  UXO  sites,  aerial  gunnery  range  munitions  (except  20  &  30mm)  are  found 
to  be  the  largest  in  mass  and  volume  while  grenade  and  submunitions  are  the  smallest.  The 
combination  of  electromagnetic  induction  and  magnetometer/gradiometer  sensors  proved  to  be 
an  effective  combination  in  all  three  survey  scenarios.  The  top  demonstrators  used  this  sensor 
technology  with  different  platforms  and  different  navigation  systems  to  detect  over  90  percent  of 


the  emplaced  ordnance.  The  ability  of  ground-based  demonstrators  to  precisely  locate  the 
ordnance  was  also  established  with  an  overall  mean  radial  error  of  0.55  meters  while  the  average 
depth  error  improved  from  0.82  meters  in  Phase  n  to  0.40  meters  in  Phase  ni.  Demonstrators’ 
size  declarations  were  correct  approximately  half  the  time,  even  with  the  implicit  size  bias  of  the 
scenario  specific  ordnance. 

To  date,  no  vendor  demonstrated  the  capability  to  discriminate  ordnance  and  non¬ 
ordnance.  This  deficiency  in  technology  would  be  expected  to  adversely  affect  UXO  site 
characterization  efforts.  In  addition,  it  would  be  a  major  cost  factor  in  any  UXO  remediation 
effort,  since  excavations  will  be  more  numerous  due  to  the  need  to  dig  up  both  ordnance  and 
non-ordnance  objects. 

The  performance  of  excavators  has  not  substantially  changed  from  Phase  I.  Each  survey 
demonstrator  reported  hundreds  of  targets  in  their  demonstration  periods  while  the  excavators 
only  unearthed  a  few  dozen  targets.  No  cost  comparison  is  offered  on  the  cost  to  detect  an  UXO 
item  versus  the  cost  to  excavate  but  it  is  apparent  that  the  two  functions  are  being  optimized  in 
isolation  from  each  other.  Disregarding  the  false  alarm  issues,  it  may  be  necessary  for  surveyors 
to  consider  efforts  that  would  improve  excavation  productivity,  such  as  centimeter  accuracy  in 
target  depth  positioning. 

Remote  excavation  is  feasible,  but  results  indicate  that  demonstrators  can  find  targets 
much  faster  than  they  can  be  excavated  using  remote  technology.  JPG  Phase  in  allowed 
technologies  to  be  defined  on  the  basis  of  their  strengths.  However,  current  UXO  technologies 
can  do  little  more  than  characterize  the  extent  of  UXO  on  properties. 

ORDNANCE  DETECTION  RATE  INCREASES  IN  JPG  PHASE  I,  II  AND  III 

Table  4  and  Graph  2  show  detection  performance  and  false  alarm  rate  for  the  first  three 
phases  of  JPG.  These  trends  only  represent  the  better  ground-based  sensor  systems  that  detected 
at  least  50  percent  of  the  baseline  ordnance  targets  at  each  of  the  phases.  The  general  trend  of 
ordnance  detection  for  succeeding  phases  is  positive.  More  industry  participants,  as  a  percentage 
of  overall  demonstrators,  are  detecting  50%  or  more  ordnance  under  the  defined  conditions. 
Average  probabilities  of  detection  increased  during  successive  phases  from  62%  to  77%. 
However,  the  false  alarm  rate  increased  from  Phase  II  to  Phase  in  after  a  significant  decrease 
between  Phases  I  and  n.  Further  work  needs  to  be  done  to  address  the  issue  of  discrimination 
and  false  alarms. 


Table  4  Ground  Based  Technology  Performance 


#  of  Ground  -Based 
Technologies* 

Pd 

False  Alarm  Rate 

Phase  I 

1  of  20 

0.62 

149 

Phase  II 

9  of  12 

0.68 

60 

Phase  III 

14  of  16 

0.77 

77 

The  first  number  is  the  number  of  ground  based  demonstrators  that  detected  over  50%  of  the  targets  while 
the  second  number  is  the  total  number  of  demonstrators  for  that  phase. 
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Graph  2  Probability  of  Ordnance  Detection  (PD)  versus  False  Alarm  Rate 


FAR  (Number  of  False  Alarms/lfetare) 


Table  5  List  of  Top  Detection  Performers  by  JPG  Phase 


Phase  I 

Phase  II 

Phase  III 

Vallon-Poof  0.65 

Geometries  -  Pd  of  0.83 

NAEVA-Pd  of  0.94 

ADI  -  Pd  of  0.48 

Parsons  -  Pd  of  0.85 

SC&A/Geometrics  -  Pd  of  0.96 

Geo-Centers  -  Pd  of  0.43 

Geophex  -  Pd  of  0.71 

Geometries  -  Pd  of  0.90 

Foerster  -  Pd  of  0.41 

ADI  -  Pd  of  0.63 

Geophex  -  Pd  of  0.77 

UXB  -  Pd  of  0.37 

Geo-Centers  -  Pd  of  0.72 

SC&A/Geo-Centers  -  Pd  of  0.76 

This  is  a  subjective  list  of  the  top  five  performers  from  each  phase.  Although 
probability  of  detection  was  the  main  focus,  false  alarm  rates  are  also  important  and  are  not 
included  for  brevity.  When  evaluating  Phase  in  results,  the  reader  should  keep  in  mind  that  the 
statistics  make  no  distinction  between  demonstrators  who  chose  to  survey  1,  2  or  all  3  of  the 
range  scenarios. 

THE  DISCRIMINATION  PROBLEM 

Phases  I,  H  and  HI  demonstrated  that  existing  magnetometers  and  electro-magnetic 
inductance  sensors  have  the  capability  to  detect  a  large  percentage  of  buried  objects,  including 
UXO.  This  can  increase  the  installation  commanders’  or  clearance  managers’  confidence  that 
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risk  can  be  reduced  to  an  aeeeptable  level.  However  the  eost  of  aehieving  an  aeeeptable  level  of 
risk  was  still  high.  The  faet  that  exeavation  of  false  alarms  due  to  system  or  environmentally 
indueed  noise  or  non-UXO  elutter  was  still  the  driving  eosts  upward.  The  solution  to 
diseriminating  between  UXO,  non-UXO  elutter  and  noise  is  not  simple.  Debate  still  eontinues  as 
to  what  defines  a  false  alarm.  Is  a  false  alarm  any  non-ordnanee  item?  The  most  useful 
diserimination  ability  would  be  between  ordnanee  items,  ordnanee-related  debris,  and  non- 
ordnanee  items.  Looking  at  the  larger  pieture,  eleanup  of  UXO  is  aetually  a  sub-set  of  the 
overall  Department  of  Defense  goal  to  reduee  risk,  irrespeetive  of  the  nature  of  that  risk. 
Providing  the  installation  eommander  or  elearanee  manager  with  the  ability  to  understand  the 
eomposition  of  all  buried  targets  provides  them  an  inereased  ability  to  understand  their  risk. 

Prior  to  Phase  IV,  teehnologies  sueh  as  advaneed  data  proeessing  and  new  sensors  types 
were  under  development  to  reduce  false  alarms  without  adversely  affeeting  ordnanee  or  risk 
deteetion  performanee.  Furthermore,  standard  sensor  data  sets  were  beeoming  publiely  available 
through  the  Defense  Advaneed  Researeh  Projeet  Ageney  (DARPA)  and  the  Joint  Unexploded 
Ordnanee  Coordination  Offiee  (JUXOCO)  to  eneourage  the  development  of  algorithms  without 
the  expense  and  burden  of  on  site  data  gathering."^  In  addition,  the  JUXOCO  has  faeilitated 
workshops  to  develop  target  diserimination  standards  and  general  guidanee  so  that  developers 
eould  exehange  and  eompare  results.  The  Department  of  Defense  and  industry  had  made  major 
investments  in  the  ability  to  diseriminate  UXO  from  elutter  and  noise  by  the  start  of  JPG-AD 
Phase  rv. 

Phase  IV 

Phase  IV  Demonstration  Design 

Phase  rv  of  the  JPG  Teehnology  Demonstration  (TD)  was  eondueted  in  1998.  Ten 
deteetion  teehnology  demonstrators  and  one  remediation  demonstrator  partieipated.  The  purpose 
of  Phase  IV  was  to: 


•  Demonstrate  the  eapabilities  of  teehnology  to  diseriminate  between  UXO  and  non- 
UXO 

•  Establish  diserimination  performanee  baselines  for  sensors  and  systems. 

•  Make  raw  sensor  data  available  to  the  publie. 

•  Establish  state-of-the-art  for  predieting  “type”  of  ordnanee. 

•  Direet  future  R&D  efforts. 

Phase  rv  also  investigated  the  eapabilities  of  various  teehnologies  to  provide  information 
about  speeifie  targets’  size,  shape,  depth,  and  orientation  underground  (previous  phases 
eoneentrated  on  deteetion). 

Both  the  40-aere  and  the  80-aere  sites  were  seeded  with  samples  of  the  same  ordnanee 
and  non-ordnanee  targets  both  with  a  ratio  of  1:2.2  respeetively  (see  Table  6).  Target  loeations 


http://www.denix.osd.mil/Public/News/UXOCOE 
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were  marked  with  flags  to  emphasize  vendor  discrimination  capabilities  rather  than  to  use  the 
allotted  time  locating  targets.  Demonstrators  who  proposed  new  data  analysis  techniques  and 
wanted  more  ground-truthed  field  data  were  given  access  to  the  80-acre  site  and  all  pertinent 
information  was  provided  on  the  seeded  targets.  This  option  allowed  demonstrators  to  calibrate 
or  train  their  systems’  response  to  known  targets.  All  demonstrators,  even  those  who  were  not 
directly  funded  to  participate  in  the  80  acre  site,  were  provided  samples  of  applicable  ordnance 
and  non-ordnance  targets  during  the  40  acre  site  demonstration. 


Table  6  Typical  ( 

Ordnance  Burial  Depths 

Diameter 

Type 

Depth  (meters) 

20mm 

projectile 

.026  to  .112 

57mm 

projectile 

.314  to  .503 

60mm 

mortar 

.214  to  .524 

76mm 

projectile 

.172  to  .559 

81mm 

mortar 

.092  to  .549 

90mm 

projectile 

.326  to  .543 

105  mm 

projectile 

.364  to  .711 

4.2” 

mortar 

.325  to  .747 

152mm 

projectile 

.579  to  1.680 

155mm 

projectile 

.680  to  .999 

Arbitrary  discrimination  performance  metrics  for  JPG  Phase  IV  were  established  based 
on  government  and  industry  input  during  the  UXO  Forum  held  in  1998.  At  that  time  desired 
goals  for  the  program  were  a  95%  true  positive  rate  with  a  75%  true  negative  rate.  True  positives 
(TP)  are  demonstrator  declared  ordnance  targets  that  are  indeed  ordnance.  True  negatives  (TN) 
are  demonstrator  declared  non-ordnance  that  is  truly  non-ordnance.  False  positives  (FP)  are 
demonstrator  declared  ordnance  that  is  actually  non-ordnance,  resulting  in  an  added  cost  burden 
to  investigate  or  remove.  False  negatives  (FN)  are  demonstrator  declared  non-ordnance  that  is 
actually  ordnance,  resulting  in  a  continuing  level  of  risk. 

Demonstrators  were  asked  to  not  only  “classify”  targets  as  ordnance  and  non-ordnance 
but  to  provide  the  government  with  the  actual  ordnance  “type”  designations  (e.g.  60mm  mortar  or 
105mm  projectile).  If  specific  information  could  not  be  provided,  “unknown”  was  an  acceptable 
demonstrator  response  for  target  type. 


Results  of  Phase  IV 

Results  of  Phase  IV  show  there  is  a  developing  capability  to  distinguish  between 
ordnance  and  non-ordnance.  Five  of  the  ten  demonstrators  showed  a  capability  to  discriminate 
ordnance  and  non-ordnance  that  was  better  than  the  chance  probability  of  50%.  One 
demonstrator  showed  a  better  than  75%  ability  to  discriminate  non-ordnance  from  ordnance 
while  maintaining  a  relatively  high  TP  rate.  The  data  generated  from  Phase  IV  was  gathered  in  a 
standard  format  and  is  available,  along  with  the  ground  truth,  to  Aided  Target  Recognition 
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(ATR)  researchers  through  the  UXOCOE  web  site.^  It  is  through  ATR  that  the  ability  to 
discriminate  will  be  provided  for  UXO  Clearance. 

Analysis  of  Phase  IV  data  is  still  underway  and  has  provided  for  lively  debate  inside  the 
UXO  Clearance  community.  The  goal  of  the  JPG-TD  program  is  not  to  give  an  absolute  answer 
but  to  show  trends  and  to  indicate  the  current  state-of-the-art.  To  help  understand  the  results  of 
Phase  rv,  Graph  3  shows  the  relationship  between  correct  declarations  of  UXO  (TP)  and  correct 
declarations  of  non-UXO  (TN).  The  line  that  runs  diagonally  from  the  top  left  to  the  lower  right 
is  the  50%  chance  line,  or  simply  guessing  about  the  target  type.  The  area  below  this  line  is 
incorrect  discrimination.  The  top  left  point  on  the  line  is  where  a  demonstrator  may  declare  all 
targets  to  be  ordnance.  Although  the  demonstrator  will  reduce  the  risk  to  zero  by  not  missing  any 
detectable  targets  he  will  have  a  larger  number  of  false  declarations  that  significantly  increase  the 
cost  of  remediation.  The  lower  right  point  indicates  where  a  demonstrator  may  declare  all  the 
targets  to  be  non-ordnance.  If  a  demonstrator  felt  that  the  vast  majority  of  target  detections  were 
non-ordnance,  he  could  declare  them  as  such.  This  scenario  would  represent  a  significantly 
higher  level  of  potential  risk.  Drawing  a  straight  line  between  these  two  points  represents  the  line 
of  “no  discrimination  ability”.  For  true  discrimination  to  take  place  a  demonstrator  must  declare 
a  combination  UXO  and  non-UXO  above  this  chance  line. 
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Graph  3  Summary  of  JPG  IV  Results 
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The  most  promising  trend  in  this  data  is  the  suceess  of  vendors  using  electro-magnetic 
inductance  equipment.  While  considered  mature  sensor  technology,  there  is  a  wealth  of 
information  in  the  frequency  and  time  domain  signals  that  provide  discrimination  capability. 

This  information  is  normally  lost  when  normal  UXO  Clearance  operators  use  the  equipment. 
However,  analysis  of  this  information  in  digital  form,  along  with  very  precise  location 
information  allows  the  use  of  ATR  algorithms  to  identify  signals  normally  ignored  by  the 
operator. 

To  further  analyze  the  data  a  comparison  of  TP,  TN,  FP,  FN  and  unknown  results  is 
needed.  Graph  4  provides  this  information  for  each  vendor.  The  graph  portrays  the  fact  that 
while  some  vendors  performed  well  in  identifying  ordnance  or  non-ordnance,  combinations  of 
sensor  technology  or  ATR  are  still  possible  that  would  move  our  discrimination  capability  further 
from  the  chance  line.  The  strength  of  one  vendor’s  approach  in  ordnance  detection  could  be 
married  with  a  vendor  strong  at  non-ordnance  to  provide  a  superior  product. 

Graph  5  shows  that  there  is  more  promise  in  some  of  the  demonstrators’  approaches.  Not 
only  did  vendors  show  a  capability  to  discriminate  UXO  from  non-UXO,  some  were  able  to 
identify  specific  types  (i.e.,  20mm  vs.  60mm)  of  UXO.  This  capability  needs  to  be  further 
explored  and  encouraged  because  it  will  provide  the  UXO  Clearance  community  with  the 
greatest  information  to  reduce  risk. 
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Graph  4  JPG  IV  Vendor  Performance 
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JPG  TD  SUMMARY 


During  the  history  of  the  JPG  Technology  Demonstrations,  76  characterization  and 
remediation  technologies  were  demonstrated.  These  efforts  provided  Government  and  Industry 
with  a  unique  method  of  learning  and  applying  knowledge  about  UXO  Clearance  in  a  controlled 
field  environment.  The  JPG  TD  allowed  the  Government  to  insert  lessons  learned  from  past 
phases  and  ongoing  UXO  Clearance  operations  to  increase  the  utility  of  the  results  for 
installation  commander  and  clearance  managers.  Industry  was  provided  an  opportunity  to  both 
demonstrate  their  capabilities  (UXO  Detection  and  Remediation  contractors)  and  understand  the 
technologies  most  applicable  to  current  UXO  Clearance  projects  (UXO  Clearance  contractors). 
JPG  TD  Phases  I  and  II  were  used  to  determine  the  emphasis  of  Phases  HI  and  IV.  Phases  I 
through  in  demonstrated  the  capabilities  of  current  technology  as  it  applies  to  the  detection  of 
ordnance  while  Phase  IV  demonstrated  the  promise  of  current  UXO  discrimination  capabilities. 
From  these  four  phases  the  following  observations  and  recommendations  can  be  made: 

•  Technology  to  detect  deeply  buried  ordnance  is  currently  inadequate.  New  sensors  need  to  be 
developed  to  detect  in-ground  targets  and  provide  higher  resolution. 

•  Current  discrimination  capabilities  need  to  be  improved. 

•  Establishment  of  test  site(s)  that  allow  industry  to  take  data  from  buried  ordnance  and  debris 
is  needed.  Documentation  should  include  target  type,  depth,  orientation,  size,  weight, 
volume,  and  type  of  material.  This  “ground  truth”  will  help  sensor  developers  focus  on  the 
specific  problems  in  ordnance  detection  along  with  the  environment  in  which  it  is  found. 

•  “Blind  tests”  or  “standards  testing”  should  be  performed  every  two  to  three  years  to  monitor 
progress  of  industry  capabilities.  Funding  for  promising  technologies  and  novel  approaches 
should  be  made  available. 

•  Combinations  of  electromagnetic  induction  and  magnetometer/gradiometer  sensors  proved  to 
be  an  effective  combination  in  all  surveys  and  should  be  developed  further. 


RECOMMENDATIONS 

Future  efforts  should  focus  on  programs  to  increase  the  true  positive  (TP)  and  true 
negative  (TN)  rates  and  to  further  characterize  “noise”  sources  on  real  ranges.  True 
identifications  decrease  both  the  risk  and  cost  of  UXO  site  remediation.  It  is  highly  desired  to 
obtain  technologies  that  have  increased  ability  to  discriminate  ordnance  from  non-ordnance,  as 
well  as  to  tell  what  type  ordnance  item  is  buried.  The  key  to  more  efficient  UXO  remediation  lies 
in  the  products  that  can  come  from  a  partnership  between  Industry  and  Government  in  an 
aggressive  development  of  cost  effective  remediation  technology  to  replace  currently  fielded 
tools  and  practices.  The  four  demonstration  phases  at  Jefferson  Proving  Ground  (JPG)  document 
some  significant  advances  in  unexploded  ordnance  (UXO)  detection,  discrimination,  and 
identification  capability  that  have  come  from  current  partnering  efforts.  The  IPG  sites  originally 
thought  to  be  simple  sites  for  the  UXO  technology  demonstrations  in  terms  of  geologic  noise  and 
cultural  clutter  backgrounds  have  characteristics  in  some  areas  that  can  make  UXO  detection 
difficult.  Detection  of  UXO  must  be  accomplished  in  the  presence  of  these  backgrounds. 
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There  are  inherent  limitations  on  the  detection  capability  of  geophysical  systems  caused 
by  the  size  and  depth  of  burial  of  UXO  (a  given  UXO  may  be  too  small  and/or  too  deep  to 
produce  a  detectable  anomaly  signature);  these  limitations  exist  regardless  of  the  geological  and 
clutter  backgrounds.  The  geological  background  further  decreases  UXO  detectability  by 
attenuating  signatures,  reducing  physical  property  contrasts,  and  providing  sources  of  localized 
anomalies.  The  cultural  background  or  clutter  decreases  the  reliability  of  UXO  detection  due  to 
interference  signals  and  false  alarm  anomalies  caused  by  surface  and  buried  cultural  features. 

It  is  strongly  recommended  that  these  technology  demonstrations  be  continued.  There  is  a 
need  to  continue  the  technology  demonstration  effort  to  document  the  capabilities  and  limitations 
of  sensor  systems  as  they  are  developed.  The  JPG  controlled  test  site  is  a  unique  national 
resource  for  assessing  UXO  technologies.  Consideration  should  be  given  to  designating  other 
DOD  areas  for  technology  demonstrations  to  provide  geological  and  vegetative  variation.  The 
Jefferson  Proving  Ground  technology  demonstrations  also  provided  a  way  for  current  UXO 
detection  systems  vendors  to  assess  their  probability  of  detection,  false-alarm  rates,  and  ability  to 
discriminate  ordnance  from  non-ordnance.  As  a  performance  test,  the  JPG  demonstrations  have 
shown  the  strengths  of  some  sensors,  the  weaknesses  of  others,  and  provided  insights  into  the 
complicated  issues  surrounding  UXO  detection  and  cleanup.  As  noted  earlier,  75%  of  UXO 
cleanup  costs  are  attributed  to  the  removal  of  non-ordnance  items.  Continued  support  of  UXO 
technology  demonstrations  will  significantly  reduce  remediation  costs  in  the  future. 
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JPG  Phase  IV,  40-Acre  Site 
Percentage  of  TP,  TN,  FP,  FN  for  All  Areas  (1,2, 3,4) 
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Note:  vendors  provided  with  x,y  position 

TP  =  (Correct  Ordnance  Declarations/TOB)  x  100 

TN  =  (Correct  Non-Ordnance  Declarations/TNOB)  x  100 

FP  =  (TNOB-TN)  X  100 

FN  =  (TOB  -TP)  X  1 00 


•^Ordnance  Declared  as  Ordnance  (TP) 
••Non-Ordnance  Declared  as  Non-Ordnance  (TN) 
Q  Non-Ordnance  Declared  as  Ordnance  (FP) 

1^  Ordnance  Declared  as  Non-Ordnance  (FN) 


JPG  Phase  IV,  40-Acre  Site 

Combined  Percentage  of  Correct  Ordnance  &  Non-Ordnance  Discriminations  (Aii  Areas) 
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